Entropy Rate Constancy in Text
نویسندگان
چکیده
We present a constancy rate principle governing language generation. We show that this principle implies that local measures of entropy (ignoring context) should increase with the sentence number. We demonstrate that this is indeed the case by measuring entropy in three different ways. We also show that this effect has both lexical (which words are used) and non-lexical (how the words are used) causes.
منابع مشابه
Entropy Converges Between Dialogue Participants: Explanations from an Information-Theoretic Perspective
The applicability of entropy rate constancy to dialogue is examined on two spoken dialogue corpora. The principle is found to hold; however, new entropy change patterns within the topic episodes of dialogue are described, which are different from written text. Speaker’s dynamic roles as topic initiators and topic responders are associated with decreasing and increasing entropy, respectively, wh...
متن کاملComputational Constancy Measures of Texts—Yule's <italic>K</italic> and Rényi's Entropy
This article presents a mathematical and empirical verification of computational constancy measures for natural language text. A constancy measure characterizes a given text by having an invariant value for any size larger than a certain amount. The study of such measures has a 70-year history dating back to Yule’s K, with the original intended application of author identification. We examine v...
متن کاملLooking for Entropy Rate Constancy in Spoken Dialog
The entropy constancy principle describes the tendency for information in language to be conveyed at a constant rate. We explore the possible role of this principle in spoken dialog, using the “summed entropy rate,” that is, the sum of the entropies of the words of both speakers per second of time. Using the Switchboard corpus of casual dialogs and a standard ngram language model to estimate en...
متن کاملAn Optimal Approach to Local and Global Text Coherence Evaluation Combining Entity-based, Graph-based and Entropy-based Approaches
Text coherence evaluation becomes a vital and lovely task in Natural Language Processing subfields, such as text summarization, question answering, text generation and machine translation. Existing methods like entity-based and graph-based models are engaging with nouns and noun phrases change role in sequential sentences within short part of a text. They even have limitations in global coheren...
متن کاملMaximum Entropy Spectral Models for Color Constancy
Earlier work showed that maximum entropy models can be used to represent surface reflectance spectra of Munsell patches. Here, we introduce a new approach to color constancy which is based upon that work. To our knowledge, all color constancy approaches employing spectral models use linear basis function representations for surface and illuminant spectra. This means that a set of basis function...
متن کامل